RapidIO is a packet based protocol. With reference to FIG. 1, there are two primary elements in a RapidIO network 1; endpoints 2 and switches 3a to 3d. Endpoints 2, such as memory, processors and bridges, can receive or transmit data packets, whereas switches 3 route the packets from endpoint 2 to endpoint 2.
A host processor 6 is defined within the RapidIO specification as the system host processor that performs a variety of system duties, such as discovery, enumeration, and initialization. In RapidIO there is typically only one system host processor 6. It may be possible to initially have two or more system hosts in RapidIO, but the first activity these system hosts must perform is determining which is the primary system host; at which point, all non-primary hosts' functions will go dormant and the primary system host 6 will perform system host duties as if it is the only host in the system. To that end, discussions within this document will refer to the “host” as the primary system host processor. Any processor in a RapidIO network can assume the role of host; there are no specific or unique hardware characteristics that distinguish a host from any other processor, only software function determines who is host.
A processing element (PE) is defined within the RapidIO specification to describe any node within a RapidIO network 1. A PE can represent the host processor 6, a bridge 7a or 7b, the switches 3a to 3d, a memory device 8, or combination thereof.
An endpoint 2 is defined within the RapidIO specification to generically describe any device, such as the processor 6, the bridge 7a or 7b, or the memory 8, which terminates the RapidIO protocol, unlike the switches 3a to 3d, which re-directs packets through the network 1.
A destination ID is defined within the RapidIO specification as an 8 bit or 16 bit unique identifier for each endpoint 2 in the network 1. Each endpoint 2 has one base destination ID register. Under normal operation, a PE will only accept packets whose header contains a destination ID value that matches the value stored in the PEs base destination ID register.
A packet comprises a header and a data payload. The header contains a variety of control information including priority, addressing, and the destination ID.
A multicast destination ID is defined within the RapidIO specification as one or more optional non-unique identifier(s) for each endpoint 2 in the network 1. A PE will only accept multicast packets, whose header contains a destination ID value that matches the value stored in any of the PEs multicast destination ID registers.
A link is defined within the RapidIO specification as a physical connection between any two PEs. A RapidIO link is a full duplex serial connection defined by a width (number of lanes) and speed (baud rate in Gbps) parameter, e.g. link 9 is defined as having 4 lanes, each with a baud rate of 3.125 Gbps.
Discovery is a process defined within the RapidIO specification wherein the host processor 6 steps through each node within the RapidIO network 1, and determines what type of device each node is, and how each node is connected to other devices in the network 1. Maintenance packets are transmitted to each PE to first determine what kind of device each PE is, e.g. switch, bridge, memory. Acknowledgement packets are returned to the host processor 6 with the initial information. Knowing the make and model of the PE, enables the host processor 6 to access basic information about the PE in its database of information for commercial devices stored in memory, but also to further interrogate the device's maintenance registers to determine other properties, e.g. routing table size, maximum number of switch ports, transaction types supported. Subsequent maintenance packets from the host processor 6 then interrogate the PE for more specific details, e.g. ports enabled. After each PE is discovered, additional maintenance packets are sent out to adjacent PE's, and the same process is repeated until the entire network is discovered, i.e. a map of the entire network with each device's connections and capabilities are stored in non-volatile memory accessible by the host processor 6.
Enumeration is a process defined within the RapidIO specification wherein the host processor 6 assigns a unique destination ID to every endpoint 2 within the network 1, which is also stored in memory accessible by the host processor 6.
Initialization is a process defined within the RapidIO specification wherein the host processor 6 initializes registers within the PEs during or following the discovery process. Enumeration is a specific type of initialization. Often initialization includes configuring routing tables within each switch 3 with the newly enumerated destination ID values, setting up various link speed or width values etc. Accordingly, for each pair of endpoints 2, a single path is determined, and the routing tables of the switches 3a to 3d therebetween are programmed to direct packets bearing a specific destination ID entering a specific input port to a specific output port.
To send a packet from one processor endpoint 2 to another in a RapidIO network, two primary requirements must be met: 1) a unique device address or destination ID (DID) for each processor endpoint 2, and 2) a pre-programmed path for the packet to flow through the switch network 1.
There is a specific register in each processor, called the based device ID register, to house the destination ID of each endpoint 2, and each switch 3a to 3d contains routing tables to guide each packet through the switch 3a to 3d using the destination ID within the packet as a lookup pointer to direct the packet to the appropriate egress port of the switch 3a to 3d. For example: to send a message from the host processor 6 to the memory endpoint 8, first, packets are generated in the host processor 6 and given destination ID 05h, i.e. the destination ID of the memory endpoint 8. The packet is then sent to the switch 3a via input port 12. Since the network 1 has already been initialized and enumerated, the switch 3a accesses its routing table and routes the packet with destination ID 05h from input port 12 to output port 0 towards the switch 3b. Similarly, the switch 3b, having received the packet at input port 4 with destination ID 05h accesses its routing table, and routes the packet from input port 4 to output port 0 towards the memory endpoint 8. Accordingly, the packet does not need to have the designated path stored therein, as the network, e.g. switches 3a to 3d, route the packets based on the destination ID, the input port, and the predetermined routing table assignments.
The RapidIO protocol has defined a packet routing mechanism that allows one endpoint 2 to send out a packet that is destined to multiple endpoints. This is similar to what other protocols term a “broadcast” with the exception that a broadcast typically sends out a packet to all endpoints whereas “multicast” by RapidIO's definition sends out a packet to a predetermined subset of all endpoints 2.
Switches 3a to 3d can be configured to associate a specific destination ID as a multicast destination ID, so when a packet enters a switch 3a to 3d with a destination ID that matches a pre-defined multicast destination ID, the packet is replicated and sent out multiple egress ports simultaneously. Multiple endpoints 2 would have one of their multicast destination ID registers set to the same value, i.e. that of the destination ID of the multicasted packet, so that multiple endpoints 2 can accept this packet. Therefore, while destination IDs assigned to endpoints 2 need to be unique, multicast destination ID registers in endpoints 2 are not unique.
To receive a multicast packet, the endpoint 2 must support one or more multicast registers, and the value of the destination ID of the packet must match the value programmed into one of the endpoints' multicast destination ID registers. However, multicast support is optional in endpoints 2, and not all endpoints 2 have multicast capabilities.
In normal operation, only packets that contain a destination ID that matches the value programmed into the endpoint's destination ID register or multicast destination ID registers, in the case of a multicast packet, will be accepted and processed by an endpoint 2. All other packets are discarded.
Some endpoints 2 support a non-standard “accept all mode”, wherein, if programmed accordingly, the endpoint 2 will accept and process any packet that reaches it, regardless of the destination ID. However, “accept all mode” is a non-RapidIO protocol feature that is supported by only some endpoint manufacturers.
Switches 3a to 3d and endpoints 2 have many maintenance registers within them that allow things such as destination IDs and routing tables to be programmed. Unlike data packets previously described, which go from endpoint 2 to endpoint 2, maintenance packets are small control packets used specifically to program device registers.
Since switches 3a to 3d, which do not have destination IDs, contain many registers, the only way to address a packet to a switch 3a to 3d according to the RapidIO specification is to use a different mechanism than what is used for a data packet. The concept of HOP count is defined in RapidIO to identify the number of devices a maintenance packet must traverse before it reaches its destination. Furthermore, the maintenance packet uses a destination ID of an endpoint 2 within the network 1 whose path happens to pass through the switch 3a to 3d being addressed so that an appropriate path for the packet can be used.
Accordingly, a maintenance packet destined for a switch, e.g. switch 3c, from the host processor 6 is provided with the destination ID, e.g. 0Fh, of an endpoint, e.g. bridge 7b, in a previously mapped path including the switches 3a and 3c and the endpoint 7b, and a HOP count, e.g. 1, based on the number of devices the maintenance packet must traverse to get to the switch, e.g. switch 3c. When the switch, e.g. switch 3a, receives the maintenance packet, it looks at the HOP count value. If it is non-zero, it decrements the HOP count value in the maintenance packet, looks up the output port, e.g. output port 4, on the routing table using the destination ID, e.g. 0Fh, of the packet, and sends the packet on its way. If the maintenance packet's HOP count is zero, the switch, e.g. switch 3c, accepts and processes the packet as if destined for itself. HOP counts are a very awkward mechanism that causes numerous problems within RapidIO, as is further explained hereinafter.
Since there are often no pre-existing routes in place following initial hardware power up, a discovery processes is initiated to find all of the devices within a system, enumerate all of the endpoints 2 with a unique destination ID, determine how they are interconnected, e.g. via switches 3a to 3d, and ultimately configure all of the routing paths from endpoint 2 to endpoint 2.
The discovery process relies on the use of maintenance packets; however, since the endpoints 2 have yet to be assigned destination IDs, no routes exist. All endpoints 2 power up with default destination IDs of 0xFF and all routing tables within switches 3a to 3d either have random data or are reset to a known value possibly 0x0. Therefore, it is impossible to utilize the normal method of an existing routing path to an endpoint 2 or through a switch 3a to 3d to address a specific device.
Instead a cumbersome method must be used of manually steering maintenance packets through a system as follows:                1) use a value of 0xFF, which is the default power up destination ID of any endpoint 2, as the destination ID in any maintenance packet;        2) use a value of 0x1 as the destination ID of the host processor 6;        3) for each switch 3a to 3d encountered, program the ingress port routing table location for destination ID 0xFF to exit the switch 3a to 3d on whichever egress port (M) is to be explored next, e.g. according to user guidance or preprogrammed order;        4) program the ingress port routing table for port (M) on switch 3a to 3d with a destination ID of 0x1 (the host processor's destination ID) so that a response packet from a potential link partner will be directed back to the host processor 6;        5) using HOP count to control how deep your maintenance packets travel into the network before reaching its destination; and        6) reprogramming routing table entries for destination ID 0xFF, so the host processor 6 can navigate maintenance packets to or through each device in a network, without relying on existing routing paths.        
The host processor 6 must keep track of each switch 3a to 3d encountered, and how the ingress and egress routing tables of every port have been programmed, so that the network 1 is accurately mapped. The host processor 6 must also enumerate each endpoint 2, and eventually go back to each switch 3a to 3d and configure the routing tables for each enumerated destination ID.
Only after the discovery process is complete can the run time approach of addressing maintenance packets using existing routes to endpoints 2 be used.
There are several deficiencies with the RapidIO protocol as it was intended to be used. For example: Dynamic Realignment, i.e. when a link fails, or transmission errors increase indicating link failure is imminent, or increased traffic on a link results in reduced throughput, it is desirable to change routes on the fly to avoid the problematic link. RapidIO does not have a mechanism that can allow for dynamic or adaptive route changes in real time. Currently, what is required in RapidIO is that: 1) transmission of data within potentially the entire system is halted; 2) routing tables in all switches are reprogrammed to include the new routing paths around the failed or failing link; and then 3) all transmission of data can be re-started
The above approach has a significant negative impact to quality of service of the system. Alternatively, a duplicate redundant system of some sort must be available, in order to perform a complete fail over to the redundant system in real time, which is how RapidIO systems today handle a link failure or significant link throughput loss while performing packet retransmissions due to errors. A redundant system is an expensive and power intensive solution, and is also a very complex programming sequence to perform, if quality of service impacts are to be minimized or eliminated. These are the common solutions used today and far from ideal as they are significantly impactful on system performance, cost and/or power.
Another shortcoming of the RapidIO protocol is the existence of isolated switches. Since maintenance registers within each switch 3a to 3d must be accessed using maintenance packets with a hop count and the destination ID of an existing endpoint 2, the path to the endpoint 2 must traverse through the switch 3a to 3d. This is often a problem as paths between endpoints 2 are usually chosen based on the shortest path to minimize transmission latencies. Therefore, it is not uncommon to have an “isolated switch”, i.e. no endpoints 2 are connected directly to the switch and no routes to other endpoints 2 go through the switch, such as the isolated switch 3d shown in FIG. 1. Accordingly, the internal registers of switch 3d would not accessible using the conventional run time addressing approach.
Isolated switches, e.g. switch 3d, present a very awkward and cumbersome mechanism to manage during hot plug events, i.e. live insertion of new boards, or potential re-routing possibilities within the system should link failures occur in other parts of the system.
The only way to access registers in isolated switches, is to use the same slow manual mechanism of steering a packet using a fixed Destination ID of 0xFF during discovery for example. Not only is this a slow and complicated process to perform simple operations such as reading registers in an isolated switch, but it has the potential of interfering with normal data flow within a network as modifications to routing tables while data is flowing through a switch is often not recommended.
A third shortcoming of the Rapid IO protocol is the inability to perform loop back testing. A very common system validation technique in networks is to allow a processor that is required to do system verification at power up, or during system operation, to send packets out through various parts of a system and then back to itself. This “loop back” method is a commonly used and simple way to validate that all paths in a system are intact and operating at peak performance.
Unfortunately, loop back testing is not possible using conventional techniques in a RapidIO network, as this would require an endpoint 2 to be capable of sending data to itself, and to not affect any valid routing of return packets from other endpoints 2. Most endpoint hardware will not allow an outgoing packet to be addressed to itself. Further, each other endpoint 2 in a system will typically have a route in place to return packets to the endpoint 2, so creating a loop using its standard destination ID will damage those existing return routing paths.
For system verification of a full mesh switch card 11, i.e. every switch 13a to 13g has a connection to every other switch 13a to 13g with only one endpoint 12, as in FIG. 2, loop back testing is impossible to do with the RapidIO protocol using conventional means as there are no other endpoints in the system and therefore no destination IDs to utilize as routing paths through any of the switches 13a to 13b. Unfortunately, the full mesh card 11 has many loops that are desired to be validated, but RapidIO protocol's limitations makes this impossible using conventional means.
An object of the present invention is to overcome the shortcomings of the prior art by providing each processing element in a RapidIO network virtual, or alternative, destination ID addresses, so that alternative paths can be dynamically reconfigured, loop-back testing can be performed, and switches can be accessed relatively easily.